Technologies for generating computer models, devices, systems, and methods utilizing the same

ABSTRACT

Technologies for generating 3D and using models are described. In some embodiments the technologies employ a content creation device to produce a 3D model of an environment based at least in part on depth data and color data, which may be provided by one or more cameras. Contextual information such as location information, orientation information, etc., may also be collected or otherwise determined, and associated with points of the 3D model. Access points to the imaged environments may be identified and labeled as anchor points within the 3D model. Multiple 3D models may then be combined into an aggregate model, wherein anchor points of constituent 3D models in the aggregate model are substantially aligned. Devices, systems, and computer readable media utilizing such technologies are also described.

FIELD

The present disclosure relates to technologies for generating computermodels. More particularly, the present disclosure relates to devices,systems, methods, and/or computer readable media for generating computermodels of real world environments, and using such models for variousapplications.

BACKGROUND

Over the past few decades video games have become increasingly popular.As a result the video game industry has grown tremendously, and nowproduces annual revenues in the United States that approach or evenexceed ten billion dollars per year. The popularity of video games hassimilarly increased in other countries, and particularly in Japan,Korea, and the European Union.

With the foregoing in mind, consumers are increasingly demanding gamesthat provide entertaining gameplay in the context of an interesting andhighly detailed graphical setting. Three dimensional (3D) computermodels of objects and environments used in video games have thereforegrown increasingly complex. In many instances such models are producedby computer graphics artists using complex 3D modeling applications.However in some instances, specialized 3D scanners may be used toconvert scale models or life size objects to 3D computer models. Ineither case, specialized hardware, software, and/or highly trainedartists may be required. Generation of 3D models of assets to be used ina complex video game and/or a virtual reality environment may thereforebe time consuming and expensive.

In some instances consumers may derive significant enjoyment from thefact that a video game or virtual reality environment is set in orotherwise simulates a real world location. This is particularly true inthe context of sports games (e.g., football, baseball, soccer, autosports, etc.). For such games, game developers often attempt toreplicate or otherwise simulate well known sporting locations such as abaseball stadium, football stadium, race track, and the like. Computergraphics artists may therefore exert painstaking effort to manuallycreate 3D models of such locations based on photographs, measurements,architectural plans, and the like. Although reasonably accurate modelsof such locations may be created in this manner, the production of suchmodels may be quite expensive and time consuming, and may still resultin a 3D model that is not a precise replica of the real world location.As a result, the generation of such models may be reserved forrelatively famous locations that may play an important role in a videogame or virtual reality experience. Put in other terms, due to the costand effort associated with the manual production of models of real worldlocations and/or objects, it may not be cost effective or otherwisedesirable to produce 3D models of less compelling real world locationsor real world objects, such as a player's real world home, a local park,generic real world housewares, and the like.

In addition to the above, video games are now routinely developed suchthat the objects and/or environments therein are designed to react in arealistic manner, e.g., in response to player interactions, simulatedweather conditions, and the like. For example virtual objects may beassociated with object properties (e.g., physical attributes such asmass, transparency, strength, etc.), such that such objects behave in arealistic or other desired manner within the game environment. However,the association of object properties to 3D models of objects and/orenvironments often entails time consuming and cumbersome manualassignment of such parameters to a plurality of individual 3D models ofsuch objects, e.g., by a computer graphics artist. Although databasesassociating pre-developed 3D models with object properties areavailable, reliance on such database(s) may undesirably limit thecreative freedom of the computer graphics artist to objects that arealready present in the database.

BRIEF DESCRIPTION OF THE DRAWINGS

Features and advantages of embodiments of the claimed subject matterwill become apparent as the following Detailed Description proceeds, andupon reference to the drawings, wherein like numerals depict like parts,and in which:

FIG. 1 depicts one example of a content creation device consistent withthe present disclosure.

FIG. 2 depicts an example of a content creation system consistent withthe present disclosure.

FIG. 3 is a diagram of example operations in accordance with one exampleof a method of generating a 3D model, consistent with the presentdisclosure.

FIGS. 4A-4D depict examples of how a plurality of 3D models may becompiled into an aggregate 3D model, consistent with the presentdisclosure.

FIG. 5 is a diagram of example operations in accordance with one exampleof a method of producing a video game with a content creation device,consistent with the present disclosure.

FIG. 6A depicts one example of a real world arrangement of threeenvironments.

FIG. 6B illustrate one example of an aggregate model of the threeenvironments of FIG. 6A arranged in a straight line.

Although the following detailed description will proceed with referencebeing made to illustrative embodiments, many alternatives,modifications, and variations thereof will be apparent to those skilledin the art.

DETAILED DESCRIPTION

While the present disclosure is described herein with reference toillustrative embodiments for particular applications, it should beunderstood that such embodiments are for the sake of example only andthat the invention as defined by the appended claims is not limitedthereto. Those skilled in the relevant art(s) with access to theteachings provided herein will recognize additional modifications,applications, and embodiments within the scope of this disclosure, andadditional fields in which embodiments of the present disclosure wouldbe of utility.

As explained in the background, computer graphics artists are oftencharged with the generation of 3D models of objects and/or environmentsthat are to be used as assets in a video game, a virtual realityenvironment, or the like. Although it is now possible to produce highlycomplex and detailed 3D models of such objects and environments, doingso may be both time consuming and expensive. This may be in part due tothe fact that in many cases, computer graphics artists produce 3D modelsusing 3D modeling software. Despite exertion of painstaking effort bysuch artists, 3D models produced in that manner may still not be ahighly accurate representation of a real world object or environment.Indeed such models may represent an object or environment with slightlydifferent dimensions, structure, texture, etc. than their real worldcounterparts.

In recent years interest has grown in the use of 3D imaging to produce3D models for video games, virtual reality applications, and the like.In that regard some effort has focused on the use of 3D imaging systemsto produce 3D models of real world objects and environments. Suchsystems may be used to “image” an environment or object so as to obtaindata which may be used to generate a 3D computer model (also referred toherein as a “computer model” or a “3D model). For example, a 3D imagingsystem may use one or more depth cameras (also known has z-camerasand/or 3D cameras) to obtain depth data from an environment/object, andutilize such depth data to produce a 3D model of the environment/objectusing techniques understood in the art. Such systems may also utilizeone or more visual (e.g., color) cameras to obtain color data from theimaged environment/object, which may be used to “texture” or “paint” the3D model produced from the depth data. The resulting output may be a 3Dmodel that reasonably simulates the structure and appearance of theimaged real-world environment/object. Indeed depending on thecapabilities of the 3D imaging system, highly accurate 3D models of realworld objects and isolated environments (e.g., single rooms) may becreated in this manner.

Although prior efforts to use 3D imaging to produce 3D models areinteresting, they may suffer from various drawbacks. For example to theinventors' knowledge existing 3D imaging systems are often limited tothe production of 3D models of an isolated environment or object, suchas a single room or a single object. Put in other terms, to theinventor′ knowledge existing 3D imaging systems may be unable to imagemultiple rooms and/or objects at the same time. Rather, to imagemultiple environments (e.g., multiple rooms), it may be necessary torelocate a 3D imaging system from one environment (room) to the next,which may be inconvenient and may introduce other issues as describedbelow.

For example existing 3D imaging systems may be used to image multipleenvironments (e.g., rooms) in the same location, e.g., by moving the 3Dimaging system such that it can image each environment in isolation ofthe other environments at the location in question. As a result,multiple discrete 3D models may be produced, with each 3D modelcorresponding to one of the imaged environments. However, to theinventors' knowledge such systems do not understand the physicalrelationship between the imaged environments and, hence, the pluralityof 3D models that are produced therefrom. That is, an existing 3Dimaging system may be unaware of the relative position of a first imagedenvironment with respect to the position of another (e.g. second) imagedenvironment in the real world, and therefore may also be unaware of thepositional relationship between 3D models of such environments. Moreoverto the inventors' knowledge, existing 3D imaging systems do not monitoror otherwise understand the movement of 3D imaging system as it is usedto image one or more environments.

Therefore while an existing 3D imaging system may be used to imagemultiple environments (e.g., rooms) to facilitate production of acorresponding number of discrete 3D models, it may be unaware of how theimaged environments are arranged relative to one another in the realworld. Therefore when such a system combines the discrete models to forman aggregate model of the imaged environments, it may arrange thediscrete 3d models such that they are in a straight line, even when theimaged environments may be arranged differently in the real world.

This issue is illustrated in FIGS. 6A and 6B. FIG. 6A depicts onehypothetical example of a real world arrangement 600 of threeenvironments, namely environments 601, 602, 603. For the sake ofclarity, it may be helpful to understand environments 601, 602, 603 asrooms, but of course any suitable type of environment may be used. Inarrangement 600, environment 602 is disposed between environments 601and 603, and includes access points 610 n, where n is an integer and inthis case is equal to 4. Although not separately labeled, environments601 and 603 include their own respective access points, which are joinedand/or shared with access points 610 n. For the sake of understanding,it may be useful to understand the access points of environments 601,602, and 603 as doors, but of course any suitable access point (e.g., awindow, hallway, etc.) may be used.

With the foregoing in mind, an existing 3D imaging system may be used toindividually “scan” or “image” each of environments 601, 602, and 603.For example, a 3D imaging system including a color (e.g. visible) cameraand a depth camera, and may be placed in the center of each ofenvironments 601, 602, 603, e.g., on a pedestal. In operation, the 3Dimaging system may use a depth camera to obtain depth information fromthe objects, walls, etc. within each environment, e.g., as the imagingsystem is rotated. Similarly, the 3D imaging system may use a visualcamera to obtain color data (e.g., color data) from each environment.

Once depth and/or color data is obtained for an environment, the 3Dimaging system may produce a distinct (i.e., separate) 3D model of eachof environments 601, 602, 603, e.g., in the manner described previously.The 3D imaging system may then be tasked with combining the distinctmodels of environments 601, 602, 603, in an effort to produce anaggregate model that reflects arrangement 600. However because the 3Dimaging system used does not understand the real world physicalrelationship (and in particular the positional relationship) betweenenvironments 601, 602, and 603, it may combine the 3D models of suchenvironments in a manner that does not accurately reflect their realworld arrangement (in this case, arrangement 600). For example and asshown in 6B, the 3D imaging system is likely to produce an aggregatemodel with an arrangement 600′, i.e., where the models 601′, 602′, and603′ are arranged in a straight line.

Moreover the 3D imaging system is likely to produce an aggregate modelthat does not take into account the relative position of the accesspoints of environments 601, 602, and 603. As a result, when the 3Dimaging system combines models 601′, 602′, and 603′, it may do sowithout regard to the alignment and/or relationship of the access pointsin such models. This concept is shown in FIG. 6B, wherein arrangement600′ includes models 603′, 602′, and 601′ that are arranged in astraight line, and one or more access points of model 602′ are notsubstantially aligned with one or more access points (not separatelylabeled) of model 601′ or model 603′, respectively.

It may therefore be appreciated that with an existing 3D imaging systemproper arrangement of models 601′, 602′, 603′ (e.g., in an aggregatemodel of the imaged environments) and their respective access points mayrequire manual intervention, e.g., by a computer graphics artists, whichmay be undesirable for at least the reasons noted above. This issue maybe exacerbated by the fact that, to the inventors knowledge, existing 3Dimaging systems do not recognize access points in the imaged environment(in the foregoing example, one or more doors), and do not associaterecognized access points with one or more points of a corresponding 3Dmodel of the environment(s) in question.

Alternatively or in addition to the foregoing issues, to the inventors'knowledge existing 3D imaging systems also facilitate the production ofa 3D model that renders or otherwise depicts an imaged environment inthe form of a single virtual object. Consider a use case in which anexisting 3D imaging system is used to image an environment (e.g., aroom) that contains a door, four walls, a ceiling, a table and chairsand a bowl on the table. Using the system, depth and color data of theenvironment may be obtained and used to produce a 3D model wherein thewalls, ceiling, table, chairs, and bowl are modeled as a single 3Dobject. That is, such systems may produce a 3D model wherein a model ofthe door is represented as integral with a model of the wall, a model ofthe table and chairs is represented as integral with the model of thefloor, etc. For example, the existing 3D imaging system may produce a 3Dmesh model of the environment wherein the mesh for the door is notdistinct from the mesh of an adjacent wall, the mesh for the table isnot distinct from the mesh of the floor, etc. In contrast, in the realworld such objects may be distinct and may be separated from oneanother.

Put in other terms, just as some existing 3D imaging systems may notunderstand the physical relationship between multiple environments, theymay also not understand the physical relationship between objects withinthose environments, between objects and boundaries (e.g., walls) ofthose environments, etc. Therefore while existing 3D imaging systems maybe able to produce an accurate representation of various objects in anenvironment, they may render such objects in a 3D model as though theywere all integral with one another, instead of as distinct objects. Itmay therefore be practically difficult to associate the various objectsin a 3D model of an environment produced by some 3D imaging systems withdistinct properties, e.g., as may be done in the development of videogames and/or virtual reality environments.

With the foregoing in mind, one aspect of the present disclosure relatesto a content creation device/system. In this regard reference is made toFIG. 1, which depicts one example of a content creation deviceconsistent with the present disclosure. In general, content creationdevice 100 may be in the form of a mobile or other electronic device.Any suitable mobile or other electronic device may be used. Non-limitingexamples of such devices include but are not limited to cameras, cellphones, computer terminals, desktop computers, electronic readers,facsimile machines, kiosks, netbook computers, notebook computers,internet devices, payment terminals, personal digital assistants, mediaplayers and/or recorders, one or more servers, set-top boxes, smartphones, tablet personal computers, ultra-mobile personal computers,wearable electronic devices (e.g., wrist worn electronic devices such asa smart watch, belt worn smart devices such as smart belt buckles, shirtworn smart devices such as smart pins, electronic headwear such as smarteyewear, combinations thereof, and the like), wired telephones,combinations thereof, and the like. Such devices may be portable orstationary. Without limitation, in some embodiments the devicesdescribed herein are in the form of a mobile electronic device such as asmart phone, a camera (e.g., a mobile 3D imaging system) a cell phone, atablet personal computer, an ultra-mobile personal computer, a wearableelectronic device, or the like. Without limitation, content creationdevice in some embodiments is in the form of a smart phone, a tabletpersonal computer, or a mobile 3D imaging system.

Regardless of its form factor and as shown in FIG. 1, content creationdevice 100 may include processor 101, memory 102, optional display 103,communications circuitry (COMMS) 104, camera module 105, one or moresensors 106, content creation module 107, and optional database (DB)109. All of such components may be communicatively coupled to oneanother via a suitable interface, such as a bus. It is noted that forthe sake of clarify FIG. 1 depicts content creation device 100 withlimited components, and with various components that may be typicallyfound in various electronic devices (e.g., antennas, multiplexers,batteries, etc., as may be found in modern mobile communicationsdevices) omitted. One of ordinary skill will understand that the omittedcomponents may be included in the architecture of content creationdevice 100 on an as needed or as desired basis. It should also beunderstood that any or all of the components of FIG. 1 may form all or apart of a device platform corresponding to the type of electronic deviceused as content creation device 100. Thus for example where contentcreation device is a smart phone, all or a portion of the components ofFIG. 1 may be present on or as a part of a smart phone platform. Incontrast where content creation device is a tablet personal computer,all or a portion of the components of FIG. 1 may be present on or aspart of a tablet personal computer platform.

It is noted that for the sake of clarity and ease of understanding, thevarious components of content creation device 100 are illustrated inFIG. 1 and are described herein as though they are part of a singleelectronic device, such as a single mobile device. It should beunderstood that such description and illustration are for the sake ofexample only, and that the various components of content creation device100 need not be incorporated into a single device. For example, thepresent disclosure envisions embodiments in which camera module 105,sensor(s) 106, CCM 107, or a combination thereof may be separate fromcontent creation device 100. For example, all or a portion of cameramodule 105 (e.g., depth camera 120, color camera 130, etc.) may form allor a part of a separate device that may be coupled in a wired orwireless manner with content creation device 100. In such instances theaggregate of such discrete components may be referred to herein as acontent creation system.

Processor 101 may be any suitable processor, such as a general purposeprocessor or application specific integrated circuit that may be capableof executing one or multiple threads on one or multiple processor cores.Without limitation in some embodiments processor 101 is a generalpurpose processor, such as but not limited to the general purposeprocessors commercially available from INTEL® Corp., ADVANCED MICRODEVICES®, ARM®, NVIDIA®, APPLE®, and SAMSUNG®. In other embodiments,processor 101 may be in the form of a very long instruction word (VLIW)and/or a single instruction multiple data (SIMD) processor (e.g., one ormore image video processors, etc.). It should be understood that whileFIG. 1 illustrates content creation device 100 as including a singleprocessor 101, multiple processors may be used.

Memory 102 may be any suitable type of computer readable memory. Examplememory types that may be used as memory 102 include but are not limitedto: semiconductor firmware memory, programmable memory, non-volatilememory, read only memory, electrically programmable memory, randomaccess memory, flash memory (which may include, for example NAND or NORtype memory structures), magnetic disk memory, optical disk memory,combinations thereof, and the like. Additionally or alternatively,memory 102 may include other and/or later-developed types ofcomputer-readable memory. Without limitation, in some embodiments memory102 is configured to store data such as computer readable instructionsin a non-volatile manner.

When used, optional display 103 may be any suitable device fordisplaying data, content, information, a user interface, etc., e.g. forconsumption and/or use by a user of content creation device 100. Thusfor example, optional display 103 may be in the form of a liquid crystaldisplay, a light emitting diode (LED) display, an organic light emittingdiode (OLED) display, a touch screen, combinations thereof, and thelike. In some embodiments, optional display 103 may be in the form of orinclude a wearable display. For example, optional display 103 may be inthe form of a display that may be worn on the head of a user, e.g., as ahelmet, eyewear, combinations thereof, or the like.

COMMS 104 may include hardware (i.e., circuitry), software, or acombination of hardware and software that is configured to allow contentcreation device 100 to receive and/or transmit data or othercommunications. For example, COMMS 104 may be configured to enablecontent creation device 100 to receive one or more contextualinformation signals from sensors 106, e.g., over a wired or wirelesscommunications link (not shown). Similarly, COMMS 104 may be configuredto enable content creation device to receive depth data, disparity data,and/or color data from camera module 105 or, more specifically, fromdepth camera 120 or color camera 130. Alternatively or additionally,COMMS 104 may enable content creation device 100 to send and receivedata and other signals to and from another electronic device, such asanother mobile or stationary computer system (e.g., a third partycomputer and/or server, a third party smart phone, a third party laptopcomputer, etc., combinations thereof, and the like). COMMS 104 maytherefore include hardware to support wired and/or wirelesscommunication, e.g., one or more transponders, antennas, BLUETOOTHchips, personal area network chips, near field communication chips,wired and/or wireless network interface circuitry, combinations thereof,and the like.

Camera module 105 may include hardware (e.g., circuitry), software, or acombination of hardware and software that is configured to allow contentcreation device 100 to image an environment/object so as to obtain atleast one of depth data and color data from the imagedenvironment/object. In this regard and as further shown in FIG. 1.,camera module may include depth camera 120 and color camera 130.

In general, depth camera 120 may include hardware (e.g., circuitry),software, or a combination of hardware and software that is configuredto obtain depth and/or color data from an imaged environment/object. Inthis regard, depth camera 120 may include a light source that isconfigured to illuminate an object/environment with output light of adesired wavelength or range of wavelengths. In some embodiments, depthcamera 120 includes a light source that is configured to output lightthat may be detected by depth camera 120, but may be invisible to colorcamera 130 and/or the human eye. Thus for example, the light source ofdepth camera 120 may emit light in a non-visible region of theelectromagnetic spectrum, such as the infrared or ultraviolet regions ofthe electromagnetic spectrum. Without limitation, the light source ofdepth camera 120 in some embodiments is configured to produce outputlight in the infrared region of the electromagnetic spectrum, i.e.,light having a wavelength between about 700 nanometers (nm) to about 1millimeter, such as about 700 to about 1400 nm.

In operation, the light source of depth camera 120 may illuminate anenvironment/object with output light. The output light may be emitted inany suitable form, such as in the form of a single spot, a pattern ofspots, a diffuse light distribution, etc. In this regard, the lightsource of depth camera 120 in some embodiments is configured to produceoutput light in a pattern which may facilitate the detection of depthdata from an illuminated environment/object. In any case depth camera120 may include one or more sensors that may detect the output lighte.g., as it impinges on and/or is reflected from an imagedenvironment/object. In some embodiments the sensor may be a lightsensor, such as but not limited to a complementary metal-oxidesemiconductor (CMOS) sensor. Without limitation in some embodiments theoutput light is infrared light, and the sensor is a CMOS infrared lightsensor. Of course, such sensors are identified for the sake of example,and any suitable sensor may be used.

In any case, the sensor(s) used in depth camera 120 may be configured todetect the output light from depth camera 120, so as to enable depthcamera 120 to obtain depth data corresponding to an imagedenvironment/object, e.g., in one or more frames. Depth camera 120 mayalso be configured to transmit one or more data signals including depthdata obtained in this manner, e.g., to content creation module (CCM)107. For ease of reference, such signals are referred to herein as depthdata signals.

As further shown in FIG. 1, camera module 105 may further include colorcamera 130. Color camera 130 may include or be in the form of hardware(e.g., circuitry), software, or a combination of hardware and softwarethat is configured to obtain color data from an imagedenvironment/object. Without limitation, color camera 130 preferably isor includes a visual camera, i.e., a camera that is designed to recordlight in the visible region of the electromagnetic spectrum. Morespecifically, color camera 130 may in some embodiments be or include avisual camera that records color data of an imaged environment/object inone or more frames.

In operation, depth camera 120 may be used to image anenvironment/object and obtain depth data in a series of first frames, asdescribed above, and to transmit such data in one or more signals (e.g.,a depth signal) to content creation module 107. In addition, colorcamera 130 may be used to image the same environment/object, so as tocapture color data from the environment/object in a series of secondframes, and to transmit such color data (e.g., in a color data signal)to content creation module (CCM) 107. Without limitation, in someembodiments CCM 107 may be configured to control the operation of depthcamera 120 and color camera 130, e.g., so as to temporally alignproduction of the first frames and second frames. That is, CCM 107 maycontrol the operation of depth camera 120 and color camera 130 such thatfirst frames of depth camera 120 are produced at substantially the sametime as second frames are produced by color camera 130. Moreover, insome embodiments CCM 107 may register first frames produced by depthcamera 120 with corresponding second frames produced by color camera130.

As noted above, camera module 105 (or, more specifically, depth camera120 and color camera 130) may transmit data signals containing depthdata and/or color data to CCM 107. While the present disclosureenvisions embodiments wherein separate data signals are used to transmitdepth data and color data (e.g., depth signals and color signals,respectively), the use of multiple signals is not required. Indeed thepresent disclosure envisions embodiments wherein camera module 105operates to transmit a combination of depth and color data in one datasignal. In such instances and for the sake of clarity, such a signal maybe referred to herein as a camera data signal.

With the foregoing in mind, content creation module (CCM) 107 mayinclude or otherwise be in the form of hardware (e.g., circuitry),software, or a combination of hardware and software that is configuredto receive depth data of an imaged environment/object (e.g., from depthcamera 120), and to create a depth map (or any other suitabledescription of depth) of the imaged environment/object from that depthdata. CCM 107 may also include or otherwise be in the form of hardware(e.g. circuitry), software, or a combination of hardware and softwarethat is configured to receive color data of an imaged environment/object(e.g., from color camera 130), and to register the color data to thedepth data, to the depth map, or a combination thereof. Alternatively oradditionally, CCM 107 may include or be in the form of a computerreadable storage medium (in which case, e.g., CCM 107 may be maintainedin memory 102) including instructions which when executed by a processor(e.g., processor 101) of content creation device 100, may cause contentcreation device 100 to perform operations consistent with the presentdisclosure, such as 3D model generation operations, aggregate modelgeneration operations, game environment generation operations,combinations thereof, and the like.

For example, in some embodiments CCM 107 may calculate registrationparameters that map the color data provided by color camera 130 to adepth map (or other representation of depth) produced at least in partfrom the depth data reported by depth camera 120. In some instances suchregistration may involve aligning points/color data of an image recordedby color camera 130 with corresponding points of depth map. CCM 107 maythen operate to create a linked list of a 2D visual (e.g. color) dataand depth values for each point in the depth map. Thereafter, CCM 107may convert the linked list to a 3D model of the imagedobject/environment, and store the 3D model on content creation device101, e.g., in memory 102. In some embodiments the 3D model is a 3D meshmodel, but of course other types of 3D models may be used.

Sensor(s) 106 are generally configured to provide contextual informationto CCM 107, e.g., in one or more contextual information signals. Ingeneral and as will be described in detail below, the contextualinformation may be used by CCM 107 to detect or otherwise determine theorientation and/or position of content creation device (e.g., as firstand second frames are produced by camera module 105) and, hence, of animaged environment and/or objects therein. In some embodiments and aswill be described later, the contextual information may also enable CCM107 to identify physical relationships between multiple imagedenvironments, multiple objects within an imaged environment, or acombination thereof.

Non-limiting examples of contextual information include locationinformation, motion information, and orientation information. In someembodiments the location information may correlate to the location ofcontent creation device 100 and/or one or more imaged objects orenvironments. One non-limiting example of such location information isglobal positioning system (GPS) coordinates specifying a location ofdevice 100 (or, more specifically, depth camera 120 and/or color camera130), e.g., when camera module 105 is used to image anenvironment/object and record first frames (containing depth data) andsecond frames (containing color data), respectively. In any case whencontextual information includes GPS coordinates, sensor(s) 106 mayinclude a GPS sensor. Of course, the use of GPS coordinates ascontextual information is for the sake of example only, and any suitablelocation information may be used. For example, in some embodiments thelocation information may correspond to the position of content creationdevice 100, e.g., relative to an item having a known location, such as aWi-Fi access point. In such instances, sensor(s) 106 may include a Wi-Fisensor that is configured to determine the location of content creationdevice 100 from Wi-Fi signals.

Motion information may reflect the motion of content creation device100, either independently or relative to an imaged environment and/orobject. One non-limiting example of suitable motion information isaccelerometer data that reflects the motion of content creation device100, e.g., as camera module produces first and/or second frames. In suchinstances, it may be understood that sensor(s) 106 may include a motionsensor, such as but not limited to an accelerometer. Of course the useof an accelerometer is for the sake of illustration only, and anysuitable motion sensor (and corresponding motion data) may be used.

Orientation information may reflect the orientation of content creationdevice 100, either independently or relative to an imaged environment orobject. As one example of suitable orientation information that may beused, mention is made of gyroscope data reflecting the orientation ofcontent creation device 100, e.g., when depth camera 120 and colorcamera record first and second frames, respectively. In such instancesit may be understood that sensor(s) 106 may include a gyroscope. Ofcourse the use of a gyroscope is for the sake of illustration only, andany suitable orientation sensor (and corresponding orientation data) maybe used.

Of course location, motion, and orientation data are merely examples oftypes of contextual information, and it should be understood that anysuitable contextual information may be used. For example, in someembodiments the contextual information includes or is in the form ofuser input data. In this context, user input data may refer to one ormore user inputs received by content creation device (e.g., through auser interface thereof). For example, user inputs could be made whencontent creation device is located at an important location within animaged environment, such as an access point (e.g., doorway, archway,etc.) thereof. As may be appreciated, such inputs may facilitateidentifying the location of various objects and/or features of an imagedenvironment, particularly when such inputs are combined with othercontextual information such as location, motion, and/or orientation dataas noted above.

With the foregoing in mind, non-limiting examples of sensors that may beused as sensor(s) 106 include a global positioning system (GPS), anaccelerometer, a gyroscope, a compass, a software sensor (e.g. to detectuser inputs), combinations thereof, and the like. Without limitation, insome embodiments, sensor(s) 106 include at least a GPS, accelerometer,and a gyroscope.

Consistent with the foregoing, in operation sensors 106 may function todetect and provide contextual information to CCM 107, e.g., in one ormore contextual information signals. In some embodiments, contextualinformation may be detected and provided as camera module 105 (or, morespecifically, depth camera 120 and/or color camera 130) is used to imagean environment and/or object. In this regard CCM 107 may be configuredto register or otherwise align contextual information of sensor(s) 106with first frames containing depth data and/or second frames containingcolor data, respectively.

For example CCM 107 may be configured to control the operation of cameramodule 105 and sensor(s) 106, such that camera module 105 collects imagedata (first frames, second frames, etc.) at substantially the same timethat sensor(s) 106 collect and provide contextual information. As CCM107 receives contextual information signals from sensor(s) 106, it mayassociate the contextual information therein with camera data (e.g.,first frames containing depth data and/or second frames containing colordata) received from camera module 105. For example, CCM 107 mayassociate contextual information with a corresponding first and/orsecond frame provided by camera module 105. In this context, acorresponding first or second frame should be understood as a firstframe or a second frame that is received by CCM 107 at substantially thesame time as CCM 107 receives the contextual information in question,e.g., from sensor(s) 106. In this regard the phrase “substantially thesame time” means within a certain threshold time period, such as fromabout 1 to about 10 seconds, milliseconds, microseconds, or anothersuitable threshold.

In any case, CCM 107 may be configured to produce or cause theproduction of a data structure that links the contextual informationprovided by sensor(s) 106 with corresponding first frames, secondframes, or a combination thereof. Without limitation, in someembodiments the data structure may be in the form of a linked list thattemporally registers or otherwise links contextual information withcorresponding first and second frames, as discussed above. As may beappreciated, CCM 107 may subsequently use such a data structure toidentify contextual information associated with a particular first orsecond frame at a particular point in time.

In instances where the contextual information is or includes locationinformation (e.g., the location of content creation device 100), CCM 107may use such contextual information (either alone or in combination withother data such as depth data or disparity data) to determine theposition of one or more objects within an imaged environment, therelative position of multiple objects in the imaged environment,combinations thereof, and the like. Moreover when the contextualinformation is or includes orientation information (e.g., theorientation of content creation device 100 at substantially the sametime that first and/or second frames were recorded), CCM 107 may usesuch information to identify or otherwise determine an orientation of animaged environment, an orientation of objects within an imagedenvironment, an orientation of content creation device 100 when depth,disparity, and/or color data was recorded, or a combination thereof. Insome embodiments, CCM 107 may utilize location information, motioninformation, and orientation information to determine the relativeposition and orientation of an imaged environment, objects within animaged environment, or a combination thereof, e.g., relative to contentcreation device 100. CCM 107 may also utilized depth data provided inone or more first frames from depth camera 120 to facilitate such adetermination.

In some embodiments, CCM 107 may assign or otherwise associate locationand/or orientation information with an imaged environment, objectswithin an imaged environment, or a combination thereof. For example andconsistent with the foregoing described, CCM 107 may produce or causethe production of a data structure such as a linked list that registerslocation and/or orientation information with corresponding points in a3D model of an environment, objects therein, or a combination thereof.Such a data structure may then be stored, e.g., as metadata inassociation with a 3D model of the environment/object produced bycontent creation device 100. As may be appreciated, such metadata may beused by CCM 107 to understand the perspective of recorded frames, theimaged environment, imaged objects, combinations thereof, and the like(e.g., relative to the content creation device at the time ofrecording), and to produce a 3D model that maintains that perspective.Moreover in some embodiments, CCM may use the metadata to understand theorientation and/or position of the imaged environment and/or objectstherein, and to produce a 3D model of the imaged environment thatmaintains that orientation and/or position.

In some embodiments CCM 107 may also include or be in the form ofhardware (e.g., circuitry), software, or a combination of hardware andsoftware that is configured to detect access points of an imagedenvironment. As used herein, the term “access point” is used to refer toa point of entry and/or egress from an imaged environment. Non-limitingexamples of access points include doorways, archways, hallways, windows,combinations thereof, and the like. Of course, other types of accesspoints are contemplated and may be used in the context of the presentdisclosure. For the sake of illustration and ease of understanding, thefollowing discussion will proceed on the basis of a use case in whichCCM 107 may detect access points in the form of one or more doors inimaged environments, namely one or more rooms. It should be understoodhowever that any suitable type of access point may be used/detected, inany suitable environment.

In this regard, as noted above CCM 107 may receive first framescontaining depth data, and second frames containing color data fromcamera module 105. In some embodiments, CCM 107 may utilize one or moreof such frames as an input image for the performance of access pointdetection operations consistent with the present disclosure. In someembodiments, such access point detection operations may include, forexample, the performance of image processing operations by CCM 107 oranother suitable component, such as process 101, a graphics processingunit (not shown) of content creation device 100, by a remote server(e.g., server 200 of FIG. 2), combinations thereof, and the like.Without limitation, in some embodiments such image processing operationsmay be or include the performance of one or more object detectionoperations on one or more input images.

For example, CCM 107 may receive first or second frames as describedabove in any suitable manner. One or more of such frames may be selectedby CCM 107 for use as an input image, e.g., for the performance of oneor more object detection operations. As may be clear, one desiredoutcome of such operations is the production of an output thataccurately identifies access points within the input image, and whichcorrelates the identified access point(s) with corresponding point(s)(e.g., coordinates, vertices, etc.) of a 3D model of an imagedenvironment.

Accordingly, CCM 107 may be configured to perform one or more objectdetection operations on an input image. CCM 107 may therefore includeone or more object detectors that may be generally configured to analyzean input image to detection regions that may contain an access point.For example, an object detector of CCM 107 may analyze or cause theanalysis of an input image for visual cues that may be indicative of anaccess point. Non-limiting examples of such visual cues include but arenot limited to structure (e.g, planes, segments, geometrical features,etc.), texture, and/or color associated with one or more pre-definedtypes of access points, such as doors, windows, archways, etc. In someembodiments, the object detector(s) within CCM 107 may give more or lessweight to one or more of the above noted factors, e.g., to compensatefor various factors that may affect the quality and/or suitability ofthe input image, such as movement of content creation device 100, noise,variable lighting, background interference, combinations thereof, andthe like. For example in some embodiments the structure of an accesspoint (as opposed to color and/or texture) may be weighted more heavilyin the analysis of an input image for access points.

With the foregoing in mind, any suitable type of object detectionparadigm may be employed by CCM 107. For example, CCM 107 may beconfigured to employ object detectors that utilize a scan stage, featureextraction stage, and classification stage to detect one or more accesspoints in an input image. In some embodiments, CCM 107 may employ anobject detector that includes or is in the form of an AdaBoostclassifier, such as a Viola-Jones detector, which can detect objects inreal-time. Without limitation, in some instances CCM 107 is configuredto utilize a Viola-Jones detector to analyze input images for featuresthat resemble Haar basis functions (hereinafter, “Haar-based features orHaar features”), and which may be correlated to an access point. Itshould be understood, however, that object detectors that analyze forother types of features (e.g., histogram oriented gradient (HOG)features, scale invariant feature transforms (SIFT), speeded up robustfeatures (SURF), oriented first rotated brief (ORB) features, localbinary patterns (LBP), planes, segments, combinations thereof, and thelike) may also be used, e.g., in combination with various machinelearning algorithms.

Regardless of the nature of the image analysis, CCM 107 may function todetect access points within an input image. When an access point isdetected, CCM 107 may be configured to identify the access point in anysuitable manner. For example CCM 107 may compare a detected access pointto a database of known types of access points, which may be storedlocally on or remote from content creation device 100. Content creationdevice 100 may therefore include an optional database (DB) 109, which inthe embodiment of FIG. 1 is illustrated as being stored in memory 102.Of course, optional DB 109 need not be stored in memory 102, and may bestored in another manner. For example, optional DB 109 in someembodiments may be stored in a remote computing system, such as but notlimited to server 200 as shown in FIG. 2. In such instances COMMS 104may be configured to enable content creation device 100 to communicatewith server 200 via a wired or wireless communications link that may beestablished using one or more pre-determined wired or wirelesscommunications protocols.

More specifically, CCM 107 may be configured to perform or cause theperformance of access point detection and identification operations. Insome embodiments, such operations may include comparing informationregarding access points detected in an input image/video to accesspoints in optional DB 109. In instances where optional DB 109 is locallystored, CMM 107 may perform a direct comparison of the access point(s)detected in an input image to the objects/access points stored in DB109. Alternatively in instances where DB 109 is stored on a remotecomputer system (e.g., server 200), CCM 107 may cause content creationdevice to transmit a data signal (object detection signal) containinginformation about access points detected in an input image/video toserver 200. The object detection signal may be configured to causeserver 200 to compare the information about access points detected inthe input image/video to objects (e.g., pre-defined access points) inoptional database 109, and to report a result to content creation device100, e.g., in an object identification signal.

In either case if the comparison reveals a match between a detectedaccess point in the input image and an access point stored in DB 109,CCM 107 may consider the detected access point to be a “true hit.” If nomatch if found, however, CCM 107 may consider the detected access pointto be a false positive and the hit (i.e., the detected access point inquestion) may be discarded. If a true hit is found, however, CCM 107 maybe configured to correlate the detected access point with acorresponding portion of a 3D model of the imaged environment. In thisregard, CCM 107 may identify depth and/or color data associated with anidentified access point, and label or otherwise identify such data ascorresponding to an access point. Likewise, CCM 107 may tag, label, orotherwise identify portions of a 3D model of an imaged environment ascorresponding to an access point. Such tags, labels, etc. may forexample be in the form of metadata, which may be associated with points(e.g., coordinates) of the 3D model corresponding to the identifiedaccess point. In some embodiments, such tags, labels, etc. may alsoidentify the type of access point, as well as the location of the accesspoint within the 3D model in question.

For the sake of clarity, points (e.g. coordinates, vertices, etc.) in a3D model that are identified as corresponding to an access point may bereferred to herein as “anchor points.” As may be appreciated, once oneor more anchor points is/are identified, CCM 107 may track the positionof such anchor points regardless of the orientation and/or modificationof the model.

While the above discussion has focused on the use of object detection toidentify access points, it should be understood that CCM 107 may beconfigured to detect and/or identify other features of an input image(e.g., a first or second frame), and to associate such features withcorresponding portions of a 3D model of an imaged environment. Forexample, CCM 107 may be configured to identify objects other than accesspoints within an imaged environment, and to associate a model of suchobjects in a 3D model of an imaged environment with the determinedidentity of the object, and optionally with suitable object properties.

The nature in which CCM 107 may identify objects in an imagedenvironment is generally the same as how it may identify access points,and therefore such operations are not reiterated herein in detail.Generally, CCM 107 (or another suitable component of content creationdevice 100) may perform image processing operations on one or more inputimages (e.g., first frames, second frames, etc.).

As before, such image processing operations may include the performanceof object detection operations on one or more input images. For example,CCM 107 may utilize an object detector to analyze an image for visualcues that may be indicative of one or more objects, such as the visualcues noted above. In any case, the image processing operations may beconfigured to account for variation factors that may affect the qualityand/or suitability of an input image under consideration. Moreover, suchobject detector of CCM 107 may use any suitable object detectionparadigm, such as those noted above. Without limitation, in someembodiments CCM 107 may employ an object detector that includes anobject classifier such as an AdaBoost classifier.

Regardless of the nature of the image processing, CCM 107 may detect oneor more within an input image. When an object is detected, CCM 107 maybe configured to identify the detected object in any suitable manner.For example, CCM 107 may compare a detected object to a databasecontaining a plurality of predefined objects, which may be storedlocally or remote from content creation device 100, e.g., in optionaldatabase 109. In such instances, optional database 109 may include aplurality of predefined objects that may be cross referenced withidentifying characteristics of such objects, and optionally with objectproperties of such objects. Non-limiting examples of such objectproperties include the dimensions of an object, physical properties ofthe object, the color of the object, combinations thereof, and the like.In any case if the comparison reveals a match is found between adetected object and an object in optional database 109, the detectedobject may be considered a true hit. Conversely if no match is found forthe detected object, the detected object may be considered a falsepositive and discarded.

If a true his found, however, CCM 107 may be configured to identifypoints (e.g., coordinates, vertices, etc.) in a 3D model of the imagedenvironment that correspond to the detected object. For example, CCM 107may identify depth and/or color data associated with an identifiedobject, and label or otherwise identify such data as corresponding tothe detected object. Likewise CCM 107 may tag, label, or otherwiseidentify portions of a 3D model of an imaged environment ascorresponding to the identified object. Without limitation, in someembodiments such tags, labels, etc. may identify the type of object withwhich they are associated. Alternatively or additionally, such tags,labels, etc. may also identify object properties associated with theobject. In such instances, CCM 107 may obtain object properties fromoptional database 109 or another suitable source, and add suchproperties to a tag, label, etc. associated with one or more portions ofa 3D model of an imaged environment, e.g., as metadata. This may allowCCM 107 to automatically assign object properties to detected objects,with the need for human involvement. As may be appreciated, this mayalleviate or even remove the need for the exertion of significant effort(e.g., by computer graphics artists) to manually associate a 3D model ofan object with one or more object properties.

In addition to identifying or causing the identification of objectswithin an imaged environment, CCM 107 may also be configured to segmentdetected objects within a 3D model of an imaged environment. In thiscontext, the terms “segment” and “segmentation” are used interchangeablyto refer to operations in which a 3D model of an object detected in animaged environment is separated from a 3D model of the imagedenvironment as a whole.

For example, consider a use case in which CCM 107 generates a 3D modelof an imaged environment that includes a model of a painting hanging ona model of a wall in a room. In that instance, CCM 107 may perform imageprocessing operations to detect and identify the painting in the imagedenvironment, and to identify the painting (and, perhaps, the wall), in a3D model of the imaged environment, as as described above. At thatpoint, CCM 107 may perform one or more segmentation operations on the 3Dmodel of the environment to separate the model of the painting from themodel of the wall, such that the painting may be represented as a 3Dmodel that is distinct from other components of the 3D model of theimaged environment, most notably the wall.

In this regard CCM 107 may employ any suitable segmentation technique tosegment identified objects within a 3D model of an imaged environment.Non-limiting examples of such techniques include shape-based retrievaltechniques, metamorphosis techniques, simplification techniques, 3Dsegmentation using a codebook model, 3D mesh segmentation,foreground-background segmentation, combinations thereof, and the like.In the painting and wall embodiment for example, one or moresegmentation techniques (e.g. foreground-background segmentation) may beused to isolate a 3D mesh model of the painting from a 3D mesh model ofthe background. The segmentation may be guided by measurements takenfrom the depth data, color data, etc., provided by camera module 105,and or which may be associated or otherwise derived from objectproperties (e.g., dimensions) stored in optional database in associatedwith a predefined object that was determined to correspond to thepainting. In any case, the result of the segmentation operation(s) maybe the production of a 3D model of the painting that is distinct fromthe remainder of the 3D model of the imaged environment, and mostnotably the wall.

In some embodiments, after one or more segmentation operations CCM 107may be configured to update the 3D model of other objects in the imagedenvironment. In the context of the painting and wall embodiment forexample, segmentation of the painting from the wall may leave a void inthe mesh model of the wall in the region behind the painting. With thisin mind, CCM 107 may perform one or more model update operations to fillin such voids or other defects, which may result from segments. Anysuitable update technique may be used for this purpose. For example, CCM107 in some embodiments may employ a fill algorithm, the execution ofwhich may fill one or more voids or other defects in a 3D model. Thefill algorithm may in some embodiments fill such voids/defects bydetermining features and/or characteristics of one or more regionsadjacent an edge of the void/defect, and filling in the void/defectbased at least in part on such features. In the painting and wallexample, for instance, CCM 107 may execute a fill or interpolationalgorithm that determines the features/characteristics of the model ofthe wall adjacent an edge of the painting, and updates the 3D model(e.g. mesh model) of the wall to fill in the region behind the paintingwith a mesh or other model having substantially the samefeatures/characteristics as the portion of the model of the walladjacent the edge of the painting.

Another aspect of the present disclosure relates to methods forproducing 3D models. In this regard reference is made to FIG. 3, whichis a flow diagram of example operations of one embodiment of a method ofgenerating a 3D model consistent with the present disclosure. As shown,model 300 begins at block 301. The method may then proceed to block 302,pursuant to which a content creation device may be used to scan/image anenvironment, such as a room. The manner in which the environment may bescanned is the same as described above in connection with the operationof content creation device 100, and therefore is not reiterated.

Once an environment has been scanned/imaged, the method may proceed fromblock 302 to block 303. Pursuant to block 303, the content creationdevice may build a 3D model of the environment based at least in part ondepth data and color data, as noted above. Points in the 3D model may beassociated with contextual information based at least in part on inputsfrom one or more sensors, e.g., location information, orientationinformation, and the like. Likewise at this stage, image processing orother operations may be performed to identify access points and to tag,label, or otherwise identify such points as anchor points in the 3Dmodel of the imaged environment.

The method may then proceed to option blocks 304, wherein objectrecognition and/or segmentation may be performed. The nature of suchoperations is described above and is therefore not reiterated in detail.If object recognition is performed, objects may be detected in theimaged environment using one or more object detection paradigms, andoperations may be performed to identify any detected objects asdescribed above. If an object is detected object and identified, themethod may further include tagging, labeling, or otherwise identifying aportion of the 3D model of the imaged environment corresponding to theidentified object with appropriate identification indicia. Furthermorethe identified object may optionally be segmented from other portions ofthe 3D model of the imaged environment, and represented as a discrete 3dmodel.

Provided object detection is performed and at least one object isdetected and identified, the method may proceed to optional block 305,wherein object properties may be associated with an identified object inthe 3D model of the imaged environment. The manner in which this may beperformed has been described previously, and thereof is not reiteratedin detail. In some embodiments, operations pursuant to block 305 mayinclude assigning object properties from a database to an identifiedobject, wherein the database is structured so as to cross reference aplurality of predefined objects with corresponding object properties.

If object recognition is not performed or object detection is performedby no objects are detected, the method may proceed from block 303 orblock 304 to block 306. Pursuant to block 306, a determination may bemade as to whether an additional environment is to be scanned/imaged. Ifso, the method may loop back to block 302. But if not, the method mayproceed to block 307, pursuant to which a determination may be made asto whether multiple environments have been scanned/imaged and modeled.If so, the method may proceed to block 308, pursuant to which anaggregate model may be compiled from a plurality of constituent 3Dmodels, as described later. If not, the method may proceed to block 309and end.

As described above, the technologies of the present disclosure mayproduce or cause the production of a 3D model of an environment, whereinall or a portion of the model is associated with contextual informationsuch as location information and/or orientation information, and whereinaccess points in the imaged environment are identified in the 3D modelthereof as one or more anchor points. With this in mind, another aspectof the present disclosure relates to technologies for producing anaggregate model from a plurality of discrete 3D models, which for thesake of clarity are referred to herein individually and collectively asa “constituent model” or “constituent models.”

As used herein, the term “aggregate model” refers to a 3D model that maybe produced by combining multiple constituent 3D models that arediscrete (i.e., separate) from one another. By way of example, thetechnologies described herein may function to combine a first(constituent) 3D model of a first environment (e.g., a first room) witha second (constituent) 3D model of a second environment (e.g., a secondroom), so as to produce an aggregate 3D model that combines the 3D modelof the first environment with the 3D model of the second environment. Ofcourse, more than 2 constituent 3D models may be combined to form anaggregate model. Indeed the present disclosure envisions embodimentswherein 2, 3, 4, 5, 10, 15, 20, 25, 100, or more constituent 3D modelsmay be combined to form an aggregate model

For the sake of illustration, the present disclosure will now proceed todescribe various embodiments in which CCM 107 operates to generate anaggregate model from two or more constituent models. It should beunderstood that this discussion is for the sake of illustration, andthat such operations need not be performed by CCM 107. Indeed thepresent disclosure envisions embodiments wherein a component other thanCCM 107 (e.g., a graphics processing unit) may be used to generate anaggregate model from a plurality of constituent 3D models.

In some embodiments, CCM 107 may function to perform one or moreaggregate model generation operations consistent with the presentdisclosure. In some embodiment such aggregate model operations mayinclude accessing at least a first constituent 3D model and a secondconstituent 3D model, analyzing the first and second constituent 3Dmodels to identify anchor points therein, and combining the firstconstituent 3D model with the second constituent 3D model such that atleast one first anchor point of the first constituent 3D model issubstantially aligned with least one second anchor point of the secondconstituent 3D model. In this context, the term “substantially aligned”may be understood to mean that greater than 95% of the points (e.g.,coordinates) of a model of a first access point in a first constituent3D model overlap with the points e.g., coordinates) of a model of asecond access point in a second constituent 3D model. Withoutlimitation, in some embodiments CCM 107 may function to produce anaggregate model in which points of one or more access points in a firstconstituent 3D model identically overlaps with points of one or moresecond access points in a second constituent 3D model.

In some embodiments, CCM 107 may utilize contextual informationassociated with anchor points of the first and second constituent 3Dmodels to facilitate combining such models such that the anchor pointssubstantially overlap. In particular, CCM 107 may analyze contextualinformation associated with the anchor points of each of the first andsecond constituent 3D models. CCM 107 may then compare the contextualinformation of each anchor point of the first constituent 3D model withthe contextual information of each anchor point of the secondconstituent 3D model and, based on that comparison, determine how thefirst and second constituent 3D models may be combined.

For example, as noted above the technologies described herein mayproduce 3D models in which location and/or orientation information maybe associated (e.g., as contextual information) with the anchor pointsof the 3D model. In such instances, CCM 107 may compare location and/ororientation information associated with each anchor point of a firstconstituent 3D model to location and/or orientation informationassociated with each anchor point of a second constituent 3D model.Based on that comparison, CCM 107 may choose to align one of the anchorpoints of the first constituent 3D models with one of the anchor pointsof the second constituent 3D models. For example in the case of locationinformation, CCM 107 may determine whether the location of any of theanchor points of a first constituent 3D model is within a thresholddistance of the location of any of the anchor points of a secondconstituent 3D model. If so, CCM 107 may combing the first and secondconstituent 3D models such that those particular anchor points aresubstantially aligned with one another. CCM 107 may then combine thefirst and second constituent 3D models, e.g., by joining them at thesubstantially aligned anchor points. For example when the first andsecond constituent 3D models are mesh models, CCM 107 may combine jointhe mesh of the first constituent 3D model with the mesh of the secondconstituent 3D model, e.g., at the substantially aligned anchor points.

When orientation information is associated with an anchor point, suchinformation may be used to facilitate the combination of multipleconstituent 3D models into an aggregate model. For example, CCM 107utilize orientation information to determine the relative orientation ofone or more anchor points in a first constituent 3D model to one or moreanchor points of a second constituent 3D model. Based on the relativeorientation of such anchor points, CCM 107 may perform a model updateoperation to one or more of the constituent 3D models, such that atleast one anchor point of the first constituent 3D model is oriented insubstantially the same direction as at least one anchor point of thesecond constituent 3D model. In such instances (and optionally guided bylocation information as discussed above), CCM 107 may combine the firstand second constituent 3D models such that the commonly oriented anchorpoint(s) of the first and second constituent 3D models are substantiallyaligned. Thereafter, model update operations may be performed to jointhe first constituent 3D model (e.g., a mesh thereof) to the (e.g., meshof the) second constituent 3D model (e.g., at the substantially alignedanchor points.

As may be appreciated, CCM 107 may have a significant amount offlexibility in determining how to align and/or combine a plurality ofconstituent 3D models. For example, CCM 107 may be configured in someembodiments to produce an aggregate model from a plurality ofconstituent 3D models, wherein the constituent 3D models arearranged/combined in the aggregate model the same manner as thearrangement of their real world counterparts. Alternatively, in someembodiments CCM 107 may be configured to produce an aggregate 3D modelfrom multiple constituent 3D models, wherein the constituent 3D modelsare arranged/combined in the aggregate model in a manner that differsfrom the arrangement of their real world counterparts. In either case,CCM 107 may function to produce an aggregate model in which one or moreaccess points of the constituent models used therein are substantiallyaligned, as discussed above.

For the purpose of clarity, the present disclosure will now describe twouse cases in which CCM 107 operates to combine a plurality ofconstituent 3D models so as to form multiple different aggregate models.With this in mind, reference is now made to FIG. 4A, which depicts oneexample of a real world arrangement of three environments, 401, 402,403. In this case environments 401, 402, 403 may be understood as rooms,but it should be understood that any suitable environment may be used.

As shown in FIG. 4A, environment 401 has a rectangular shape with a longdimension running left to right. A single access point (e.g., a door)410 is present in the right wall of environment 401. In contrast,environment 402 has a rectangular shape with a long dimension runningfrom top to bottom. Moreover, multiple access points 420 ₁-420 ₄ arepresent in environment 402, with one access point being located withineach wall thereof. Environment 403 is similar to environment 401, exceptthat it includes an access point (door) 430 that is formed in its leftwall at a location that differs from that of access point 410 ofenvironment 401.

With the foregoing in mind, content creation device 100 may be employedto generate distinct 3D models of each of environments 401, 402, and403, as generally described above. That is, camera module 105 may beused to collect depth and color data of each of environments 401, 402,403, e.g., as content creation device 100 is moved within eachenvironment. At substantially the same time, contextual information(e.g., location information, motion information etc.) may be collectedby sensor(s) 106. The depth data, color data, and contextual informationmay then be provided to CCM 107, which may use such data to produceconstituent 3D models 401′, 402′, 403′, as illustrated in FIG. 4B and inthe manner previous described. CCM 107 may then perform access pointdetection operations on input images (e.g. depth data, sensor data,etc.) and/or constituent 3D models 401′, 402′, 403′, resulting in theidentification of access points 410, 420 ₁₋₄, and 430. CCM 107 may thentag, label, or otherwise identify the identified access points as anchorpoints 410′, 420 ₁₄′, and 430′, as shown in FIG. 4B. After that point,CCM 107 may perform aggregate model generation operations to combinemodels 401′, 402′, 403′, as generally described above.

For example, in some embodiments CCM 107 may be configured to generatean aggregate model simulating the real world arrangement of environments401, 402, and 403, based at least in part on a comparison of contextualinformation associated with each of the anchor points identified inmodels 401′, 402′, and 403′. In this embodiment for example, CCM 107 maydetermine that location information associated with anchor point 410′ ofmodel 401′ is within a threshold distance of the anchor point 420 ₁′ ofmodel 402′. In such instances, CCM 107 may then substantially alignanchor point 410′ with anchor point 420 ₁′, and perform model updateoperations to join (e.g., the mesh of) model 401′ with model 402′ atthose anchor points. In some embodiments, CCM 107 may align anchorpoints among two or more constituent 3D models based at least in in parton a comparison of orientation information associated with such anchorpoints. For example, CCM 107 may analyze orientation informationassociated with two or more anchor points of differing constituent 3Dmodels to determine whether the anchor points are oriented in the samedirection. If the two anchor points are commonly oriented, CCM 107 mayproceed to align the anchor points by bringing them into close proximitywith one another, such that they touch or overlap one another. If thetwo anchor points are not commonly oriented, however, CCM 107 mayreorient one or both of the constituent 3D models in question relativeto other to compensate for any difference in the orientation of theconstituent 3D models, after which the models may be aligned asdescribed above.

Returning to the illustrative embodiment, CCM 107 may perform similaroperations to identify that anchor points 420 ₃′ and 430′ are within athreshold distance, and/or are aligned in substantially the samedirection. CCM 107 may then substantially align anchor point 420 ₃′ withanchor point 430′, and perform model update operations to join (e.g.,the mesh of) model 402′ with model 403′ at those anchor points. Theresult may be the production of an aggregate model 450 in which models401′, 402′, and 403′ are arranged in the same manner as environments401, 402, and 403, with the models being joined to one another at therelevant access points, as shown in FIG. 4C.

In other embodiments CCM 107 may be configured to generate an aggregatemodel from a plurality of constituent 3D models, wherein the constituent3D models in the aggregate model are arranged in a manner that differsfrom their real world counterparts. As in the prior example embodiment,content control device may be used to image environments 401, 402, 403and produce constituent models 401′, 402′, 403′ thereof. All or aportion of the points (e.g. vertices) of models 401′, 402′, 403′ may beassociated with contextual information, object properties, and the like,as previously described. Moreover, anchor points in each model may betagged or otherwise identified in each model, as previously described.

With the foregoing in mind, CCM 107 may analyze each constituent 3Dmodel to be included in the aggregate model to identify the anchorpoints therein. Thus in this example, CCM 107 may analyze metadata,tags, etc. associated with the anchor points of each of models 401′,402′, and 403′. CCM 107 may then combine any two anchor points fromdifferent constituent 3D models. For example, CCM 107 may select oneanchor point of model 402′ to join with anchor point 430′ of model 403′,and another anchor point of model 402′ to join with anchor point 420′ ofmodel 402′.

The manner in which the anchor points may be selected for combination isnot limited, and may be accomplished in any manner. For example, anchorpoints may be selected based on a user profile, an algorithm, the outputof a random number generator, the relative size of the constituentmodels to be included in the aggregate model, the orientation of suchmodels and/or the objects contained therein, combinations thereof, andthe like. In some embodiments, CCM 107 may function to select anchorpoints for combination based at least in part on the geometry of theconstituent 3D models to be included in the aggregate model, theorientation of the constituent 3D models, the orientation of identifiedobjects in the constituent 3D models, combinations thereof, and thelike. Without limitation, CCM 107 in some embodiments selects anchorpoints for combination such that one constituent 3D model does notoverlap or otherwise conflict with another constituent 3D model. CCM 107may then align the selected anchor points and join their respectiveconstituent 3D models in much the same manner as noted above. Notably,CCM 107 or another component of content creation device 100 may reorientor otherwise rearrange one or more or the constituent 3D models and/orobjects therein to facilitate the combination of the selected anchorpoints.

To illustrate this concept reference is made to FIG. 4D, which depictsan aggregate model that is constructed from the same constituent 3Dmodels as the aggregate model of FIG. 4C, but wherein the constituent 3Dmodels are arranged in a manner that differs from their real worldcounterparts. More specifically, in this embodiment CCM 107 has selectedanchor point 430′ of model 403′ for combination with anchor point 420_(k)′ of model 402′, and anchor point 420 ₄ of model 402′ forcombination with anchor point 410′ of model 401′ for combination. As maybe appreciated, the orientation of models 401′ and 403′ in FIG. 4D isdifferent from the orientation of such models in FIG. 4B. It maytherefore be understood that prior to combining such models with model402′ at the selected anchor points, CCM 107 may have reoriented orotherwise rearranged models 401′, 402′, and/or 403′ relative to oneanother, until the selected anchor points of each model are orientedproperly with regard to one another. At that point, CCM 107 may alignand join the models in substantially the same manner as previouslydescribed.

As described above, the technologies of the present disclosure mayfacilitate the production of 3D models of real world environments. Such3D models may be used independently for a variety of purposes, or theymay be combined to form one or more aggregate models. As may beappreciated, such models may have applications in a wide variety of usecases, such as but not limited to the development of virtual spaces foruse in video games, virtual reality experiences, combinations thereof,and the like. Alternatively or additionally, because the 3D modelsdescribed herein may associate contextual information, objectproperties, etc., they may also have use in virtual histories, e.g.,where a history of a particular object may be recorded/developed frommultiple scans/3D models of the same environment developed over time.

In the context of video games and virtual devices, for example, thecontent creation devices described herein may be employed to generate 3Dmodels of a real world location, such as but not limited to a user'shome. For example, a content creation device may be used to image eachroom in a user's home, and then generate a corresponding 3D model ofsuch rooms. As generally described above, contextual information such aslocation information, orientation information, etc., may be associatedwith all or a portion of the 3D model. In addition, objects within theimaged environment may be identified, segmented, and autonomouslyassociated with relevant object properties, e.g., by retrieving suchproperties from a local or remotely stored database, and associatedthose properties with segmented objects within the 3D model of a room.

Consistent with the foregoing description, a content creation module maythen be used to generate an aggregate model of the imaged environments,wherein the aggregate model accurately reflects the arrangement ofrooms, spaces, etc., in the imaged environment, in this case a user'shome. Alternatively, the content creation module could be used toproduce an aggregate model in which the models of the rooms are arrangedin a manner that differs from the arrangement of real worldcounterparts. In such instances, the arrangement of the modeled roomsmay change each time the aggregate model is generated, e.g., so as toprovide a dynamic experience for the user each time the aggregate modelis used as a game environment.

Turning now to FIG. 5, another aspect of the present disclosure relatesto generating and modifying an aggregate model, e.g., for use as a videogame environment. As shown the method 500 begin at block 501. If a 3Dmodel consistent with the present disclosure has not been generated, themethod may proceed to optional block 500, pursuant to which the modelgeneration method of FIG. 3 may be performed. Once a 3D model has beengenerated (or if a 3D model is previously available), the method mayproceed to block 503.

Pursuant to block, game characters may be introduced into the 3D model,e.g., by CCM 107, another component of content creation device 107,and/or by a component of a remote device, such as server 200. Anysuitable game characters may be used, as generally understood in theart. Alternatively or additionally, game characters may be derived fromobjects, persons, pets, etc. imaged by the content creation device,e.g., during the creation of all or a portion of the 3D model used asthe video game world. For example where a user's dog is present in aroom when it is imaged by a content creation device consistent with thepresent disclosure, the dog may be identified, e.g., by the user or viathe execution of object/image recognition operations. A 3D model of theimaged entity (in this case, the dog) may then be created and segmented(if necessary) from the other elements of the 3D model.

The 3D model of the imaged entity may then be stored local or remotelyfrom content creation device 100, e.g., in character store. Thecharacter store may be present, for example, as a data structure storedin a memory of content creation device 100, server 200, or in anothersuitable location. Furthermore while the foregoing description focusedon the generation of one character model from an image of a dog, itshould be understood that any number of character models may begenerated and stored in the character store. Moreover it should beunderstood that the characters in the character store need not bederived from images captured by content creation device 100, and may begenerated in another manner, such as by a computer graphics artist usingtraditional means.

In any case, CCM 107 or another component of content creation device 100may insert characters from the character store into a 3D model (e.g., anaggregate model), to provide a desired user experience. For example, CCM107 or another component of content creation device 100 may insertcharacters from the character store to serve as non-player characters(e.g., allies, adversaries, etc.), which may be interacted with by auser of content creation device 100.

Returning to FIG. 5, once game characters are introduced the method mayproceed to element 504, pursuant to which the player's location may betracked. In this regard, it may be understood that content creationdevice may form all or part of a wearable device that may be used toprovide a gaming experience to a user. For example the 3D model used asthe game world may be rendered on optional display 103, and may changeas a user moves with content creation device 100. Put in other terms, ininstances where the 3D model is of a user's home, the user may interactwith that model by moving with content creation device throughout hishome. As the user (or, more specifically, content creation device 100)moves, the 3D model of the user's home may be displayed and updated onoptional display 103. GPS or other location sensors (e.g., WIFI sensors)may be used to track the player's location within the player's home, andupdate the rendering of the 3D model on optional display 103accordingly.

The method may then proceed to block 505, pursuant to which the 3Dmodel/game characters may be updated, e.g., in response to userinteraction. For example, a user may interact with the 3d model/gamecharacters via a graphical user interface, one or more interface devices(e.g., game controllers), combinations thereof, and the like. The resultof such interaction may be the alteration of the 3D model, a state ofone or more game characters, etc. Pursuant to method 500, CCM 107,another component of content creation device 100, or a component ofserver 200 may update the 3D model, game characters, etc., in responseto the user interaction, so as to provide a desired user experience.

The method may then proceed to block 506, pursuant to which adetermination may be made as to whether the method is to continue. Ifso, the method may loop back to block 503, 504, or 505, as desired. Butif not, the method may proceed to block 507 and end.

EXAMPLES

The following examples enumerate additional non-limiting embodiments ofthe present disclosure.

Example 1

According to this example there is provided content creation deviceincluding a processor, a memory, a camera module, and a content creationmodule (CCM), wherein: the camera module includes a depth camera toobtain depth data of a real world environment and a visual camera toobtain color data of the real world environment; and the CCM includescircuitry to: produce a three dimensional model of the real worldenvironment based at least in part on the depth data; analyze at leastone of the depth data and color data to detect one or more access pointsin the real world environment; and correlate a portion of the threedimensional model with a detected access point.

Example 2

This example includes any or all of the features of example 1, andfurther includes at least one sensor to obtain contextual information,wherein the CCM is further to correlate the contextual information withthe portion of the three dimensional model corresponding to a detectedaccess point.

Example 3

This example includes any or all of the features of example 2, whereinthe at least one sensor includes one or more of a gyroscope, a globalpositioning sensor, or an accelerometer.

Example 4

This example includes any or all of the features of example 3, wherein:the at least one sensor is a global positioning sensor; and thecontextual information includes location information.

Example 5

This example includes any or all of the features of any one of examples1 to 4, wherein: the depth data is stored in one or more first frames inthe memory; the color data is stored in one or more second frames in thememory; and the CCM is to detect the one or more access points at leastin part by executing image processing operations on at least one of thefirst or second frames.

Example 6

This example includes any or all of the features of example 5, whereinthe CCM is further to identify a type of the detected access point.

Example 7

This example includes any or all of the features of example 6, wherein:the image processing operations comprise performing object detection onat least one of the first or second frames; the CCM is further toidentify the type of the detected access point at least in part bycomparing an output of the object detection operations to a database ofpredefined access points.

Example 8

This example includes any or all of the features of example 7, whereinthe database of predefined access points is stored in the memory ormaintained remotely from the content creation device.

Example 9

This example includes any or all of the features of any one of examples1 to 4, wherein: the CCM is further to analyze at least one of depthdata and color data to detect an object within the real worldenvironment; and the three dimensional model includes a representationof a detected object within the real world environment.

Example 10

This example includes any or all of the features of example 9, wherein:the depth data is stored in one or more first frames in the memory; thecolor data is stored in one or more second frames in the memory; and theCCM is to detect the objects at least in part by executing imageprocessing operations on at least one of the first or second frames.

Example 11

This example includes any or all of the features of example 10, whereinthe CCM is further to identify a detected object in the real worldenvironment.

Example 12

This example includes any or all of the features of example 11, wherein:the image processing operations comprise performing object detection onat least one of the first or second frames; and the CCM is to identify adetected object at least in part by comparing an output of the imageprocessing operations to a database of predefined objects.

Example 13

This example includes any or all of the features of example 11, wherein:the CCM is further to determine object parameters corresponding to anidentified detected object; and correlate the object parameters with arepresentation of an identified detected object in the three dimensionalmodel.

Example 14

This example includes any or all of the features of example 11, whereinthe CCM is to determine the object parameters based at least comparisonof an identified object to a database correlating a plurality ofpredefined objects with corresponding object parameters.

Example 15

This example includes any or all of the features of example 11, whereinthe CCM is further to perform segmentation operations on the threedimensional model to segment a representation of the identified objectin the three dimensional model from the remainder of the threedimensional model.

Example 16

This example includes any or all of the features of example 15, whereinthe segmentation operations result in a void in the three dimensionalmodel, and the CCM is further to perform model update operations to fillin the void.

Example 17

This example includes any or all of the features of any one of examples1 to 4, wherein: the CCM is further to combine a plurality ofconstituent three dimensional models to produce an aggregate model, theplurality of constituent three dimensional models including a firstconstituent model of a first real world location and a secondconstituent model of a second real world location, the first and secondreal world locations being arranged in a real world arrangement; and inthe aggregate model, the first and second constituent models arearranged in substantially the same manner as the real world arrangement.

Example 18

This example includes any or all of the features of example 17, wherein:the first and second constituent models comprise first and second meshmodels, respectively; and the CCM is further to combine the plurality ofconstituent three dimensional models at least in part by combining amesh of the first mesh model with a mesh of the second mesh model.

Example 19

This example includes any or all of the features of example 17, wherein:the first constituent model is associated with first metadataidentifying one or more first anchor points in the first constituentmodel, each first anchor point corresponding to an access point in thefirst real world location; the second constituent model is associatedwith second metadata identifying one or more second anchor points in thesecond constituent model, each second anchor point corresponding to anaccess point in the first real world location; and the CCM is further toproduce the aggregate model at least in part by joining at the first andsecond constituent models, such that in the aggregate model a firstanchor point of the first constituent model is substantially alignedwith a second anchor point of the second constituent model.

Example 20

This example includes any or all of the features of example 19, wherein:the first constituent model includes a plurality of first anchor points;and the CCM is further to combine the plurality of constituent modelssuch that in the aggregate model, a selected one of the plurality offirst anchor points is substantially aligned with a second anchor pointof the second constituent model.

Example 21

This example includes any or all of the features of example 20, wherein:each of the plurality of first anchor points of the first constituentmodel is associated with first contextual information; each secondanchor point of the second constituent model is associated with secondcontextual information; and the CCM is further to identify the selectedone of the plurality of first anchor points based at least on acomparison of the first contextual information of each of the pluralityof first anchor points with second contextual information associatedwith each second anchor point.

Example 22

This example includes any or all of the features of example 21, whereinthe contextual information includes location information.

Example 23

This example includes any or all of the features of example 22, whereinthe location information specifies a real world location of each of thefirst and second anchor points.

Example 24

This example includes any or all of the features of example 23, whereinthe CCM is further to identify the selected one of the plurality offirst anchor points based at least in part on a determination that afirst anchor point is within a threshold distance of a second anchorpoint.

Example 25

This example includes any or all of the features of example 21, whereinthe contextual information further includes orientation information, theorientation information specifying an orientation of a corresponding oneof the first and second anchor points.

Example 26

This example includes any or all of the features of example 25, whereinthe CCM is to identify the selected one of the plurality of first anchorpoints based at least in part on a determination that a first anchorpoint is within a threshold distance of a second anchor point.

Example 27

This example includes any or all of the features of example 26, whereinthe CCM is further to determine whether the orientation of the selectedone of the plurality of first anchor points is the same as the secondanchor point and, if not, to reorient at least one of the firstconstituent model and the second constitute model.

Example 28

This example includes any or all of the features of example 21, whereinthe CCM is to identify the selected one of the plurality of first anchorpoints such that in the aggregate model, one or more of the plurality ofconstituent three dimensional models does not conflict with another ofthe plurality of constituent three dimensional models.

Example 29

This example includes any or all of the features of example 17, wherein:the aggregate model is a video game world; and the CCM is to introducevideo game characters into the video game world.

Example 30

According to this example there is provided a method for producing athree dimensional model, including, with a content creation device:producing a three dimensional model of a real world environment based atleast in part on depth data of the real world environment; analyzing atleast one of the depth data and color data of the real world environmentto detect one or more access points in the real world environment; andcorrelating a portion of the three dimensional model with a detectedaccess point.

Example 31

This example includes any or all of the features of example 30, whereinthe method further includes correlating contextual information of atleast one sensor with the portion of the three dimensional modelcorresponding to a detected access point.

Example 32

This example includes any or all of the features of example 31, whereinthe contextual information includes at least one of orientationinformation, location information, or accelerometer information.

Example 33

This example includes any or all of the features of example 30, whereinthe contextual information includes location information.

Example 34

This example includes any or all of the features of any one of examples30 to 33, wherein the method further includes obtaining the depth datain one or more first frames with a depth camera; and obtaining the colordata with one or more second frames with a visual camera; wherein theanalyzing at least one of the depth data and color data includesexecuting image processing operations on at least one of the first orsecond frames.

Example 35

This example includes any or all of the features of example 34, whereinthe method further includes identifying a type of the detected accesspoint.

Example 36

This example includes any or all of the features of example 35, wherein:the image processing operations comprise performing object detection onat least one of the first or second frames, and the identifying isperformed at least in part by comparing an output of the objectdetection operations to a database of predefined access points.

Example 37

This example includes any or all of the features of example 36, whereinthe database of predefined access points is stored in a memory of thecontent creation device or is maintained remotely from the contentcreation device.

Example 38

This example includes any or all of the features of any one of examples30 to 33, wherein the method further includes analyzing at least one ofthe depth data and color data to detect an object within the real worldenvironment; and the three dimensional model includes a representationof a detected object within the real world environment.

Example 39

This example includes any or all of the features of example 38, whereinthe method further includes obtaining the depth data in one or morefirst frames with a depth camera; and obtaining the color data with oneor more second frames with a visual camera; wherein the analyzing atleast one of the depth data and color data to detect an object includesexecuting image processing operations on at least one of the first orsecond frames.

Example 40

This example includes any or all of the features of example 39, whereinthe method further includes identifying a detected object in the realworld environment.

Example 41

This example includes any or all of the features of example 40, wherein:the image processing operations comprise performing object detection onat least one of the first or second frames; and the identifying adetected object includes at least in part by comparing an output of theimage processing operations to a database of predefined objects.

Example 42

This example includes any or all of the features of example 40, whereinthe method further includes: determining object parameters correspondingto an identified detected object; and correlating the object parameterswith a representation of an identified detected object in the threedimensional model.

Example 43

This example includes any or all of the features of example 40, whereinthe determining the object parameters is performed at least in part bycomparing an identified object to a database correlating a plurality ofpredefined objects with corresponding object parameters.

Example 44

This example includes any or all of the features of example 40, whereinthe method further includes: executing segmentation operations on thethree dimensional model to segment a representation of the identifiedobject in the three dimensional model from the remainder of the threedimensional model.

Example 45

This example includes any or all of the features of example 44, whereinthe segmentation operations result in a void in the three dimensionalmodel, and the CCM is further to perform model update operations to fillin the void.

Example 46

This example includes any or all of the features of any one of examples30 to 33, wherein the method further includes, with the content creationdevice: combining a plurality of constituent three dimensional models toproduce an aggregate model, the plurality of constituent threedimensional models including a first constituent model of a first realworld location and a second constituent model of a second real worldlocation, the first and second real world locations being arranged in areal world arrangement; wherein in the aggregate model, the first andsecond constituent models are arranged in substantially the same manneras the real world arrangement.

Example 47

This example includes any or all of the features of example 46, wherein:the first and second constituent models comprise first and second meshmodels, respectively; and combining the plurality of constituent threedimensional models includes combining a mesh of the first mesh modelwith a mesh of the second mesh model.

Example 48

This example includes any or all of the features of example 46, wherein:the first constituent model is associated with first metadataidentifying one or more first anchor points in the first constituentmodel, each first anchor point corresponding to an access point in thefirst real world location; the second constituent model is associatedwith second metadata identifying one or more second anchor points in thesecond constituent model, each second anchor point corresponding to anaccess point in the first real world location; and combining theplurality of constituent three dimensional models to produce theaggregate model includes joining the first and second constituent modelssuch that in the aggregate model, a first anchor point of the firstconstituent model is substantially aligned with a second anchor point ofthe second constituent model.

Example 49

This example includes any or all of the features of example 48, wherein:the first constituent model includes a plurality of first anchor points;and in the aggregate model, a selected one of the plurality of firstanchor points is substantially aligned with a second anchor point of thesecond constituent model.

Example 50

This example includes any or all of the features of example 49, wherein:each of the plurality of first anchor points of the first constituentmodel is associated with first contextual information; each secondanchor point of the second constituent model is associated with secondcontextual information; and the method further includes identifying theselected one of the plurality of first anchor points based at least on acomparison of the first contextual information of each of the pluralityof first anchor points with second contextual information associatedwith each second anchor point.

Example 51

This example includes any or all of the features of example 50, whereinthe contextual information includes location information.

Example 52

This example includes any or all of the features of example 51, whereinthe location information specifies a real world location of each of thefirst and second anchor points.

Example 53

This example includes any or all of the features of example 52, whereinidentifying the selected one of the plurality of first anchor pointsincludes determining whether a first anchor point is within a thresholddistance of a second anchor point.

Example 54

This example includes any or all of the features of example 50, whereinthe contextual information further includes orientation information, theorientation information specifying an orientation of a corresponding oneof the first and second anchor points.

Example 55

This example includes any or all of the features of example 54, whereinidentifying the selected one of the plurality of first anchor pointsincludes determining whether a first anchor point is within a thresholddistance of a second anchor point.

Example 56

This example includes any or all of the features of example 55, whereinthe method further includes determining whether the orientation of theselected one of the plurality of first anchor points is the same as thesecond anchor point and, if not, reorienting at least one of the firstconstituent model and the second constitute model.

Example 57

This example includes any or all of the features of example 50, whereinidentifying the selected one of the plurality of first anchor points isperformed such that in the aggregate model, one or more of the pluralityof constituent three dimensional models does not conflict with anotherof the plurality of constituent three dimensional models.

Example 58

This example includes any or all of the features of example 46, wherein:the aggregate model is a video game world; and the method furtherincludes introducing video game characters into the video game world.

Example 59

According to this example there is provided at least one computerreadable medium including instructions which when executed by aprocessor of a content creation device result in the performance of thefollowing operations including: producing a three dimensional model of areal world environment based at least in part on depth data of the realworld environment; analyzing at least one of the depth data and colordata of the real world environment to detect one or more access pointsin the real world environment; and correlating a portion of the threedimensional model with a detected access point.

Example 60

This example includes any or all of the features of example 59, whereinthe instructions when executed further result in the performance of thefollowing operations including: correlating contextual information of atleast one sensor with the portion of the three dimensional modelcorresponding to a detected access point.

Example 61

This example includes any or all of the features of example 60, whereinthe contextual information includes at least one of orientationinformation, location information, or accelerometer information.

Example 62

This example includes any or all of the features of example, wherein thecontextual information includes location information.

Example 63

This example includes any or all of the features of any one of examples59 to 62, wherein the instructions when executed further result in theperformance of the following operations including: obtaining the depthdata in one or more first frames with a depth camera; and obtaining thecolor data with one or more second frames with a visual camera; whereinthe analyzing at least one of the depth data and color data includesexecuting image processing operations on at least one of the first orsecond frames.

Example 64

This example includes any or all of the features of example 63, whereinthe instructions when executed further result in the performance of thefollowing operations including: identifying a type of the detectedaccess point.

Example 65

This example includes any or all of the features of example 64, wherein:the image processing operations comprise performing object detection onat least one of the first or second frames, and the identifying isperformed at least in part by comparing an output of the objectdetection operations to a database of predefined access points.

Example 66

This example includes any or all of the features of example 65, whereinthe database of predefined access points is stored in a memory of thecontent creation device or is maintained remotely from the contentcreation device.

Example 67

This example includes any or all of the features of any one of examples59 to 62, wherein the instructions when executed further result in theperformance of the following operations including: analyzing at leastone of the depth data and color data to detect an object within the realworld environment; and the three dimensional model includes arepresentation of a detected object within the real world environment.

Example 68

This example includes any or all of the features of example 67, whereinthe instructions when executed further result in the performance of thefollowing operations including: obtaining the depth data in one or morefirst frames with a depth camera; and obtaining the color data with oneor more second frames with a visual camera; wherein the analyzing atleast one of the depth data and color data to detect an object includesexecuting image processing operations on at least one of the first orsecond frames.

Example 69

This example includes any or all of the features of example 68, whereinthe instructions when executed further result in the performance of thefollowing operations including: identifying a detected object in thereal world environment.

Example 70

This example includes any or all of the features of example 69, wherein:the image processing operations comprise performing object detection onat least one of the first or second frames; and the identifying adetected object includes at least in part by comparing an output of theimage processing operations to a database of predefined objects.

Example 71

This example includes any or all of the features of example 69, whereinthe instructions when executed further result in the performance of thefollowing operations including: determining object parameterscorresponding to an identified detected object; and correlating theobject parameters with a representation of an identified detected objectin the three dimensional model.

Example 72

This example includes any or all of the features of example 69, whereinthe determining the object parameters is performed at least in part bycomparing an identified object to a database correlating a plurality ofpredefined objects with corresponding object parameters.

Example 73

This example includes any or all of the features of example 69, whereinthe instructions when executed further result in the performance of thefollowing operations including: executing segmentation operations on thethree dimensional model to segment a representation of the identifiedobject in the three dimensional model from the remainder of the threedimensional model.

Example 74

This example includes any or all of the features of example 73, whereinthe segmentation operations result in a void in the three dimensionalmodel, and the instructions when executed further result in theperformance of the following operations including: performing modelupdate operations to fill in the void.

Example 75

This example includes any or all of the features of any one of examples59 to 62, wherein the instructions when executed further result in theperformance of the following operations including: combining a pluralityof constituent three dimensional models to produce an aggregate model,the plurality of constituent three dimensional models including a firstconstituent model of a first real world location and a secondconstituent model of a second real world location, the first and secondreal world locations being arranged in a real world arrangement; whereinin the aggregate model, the first and second constituent models arearranged in substantially the same manner as the real world arrangement.

Example 76

This example includes any or all of the features of example 75, wherein:the first and second constituent models comprise first and second meshmodels, respectively; and combining the plurality of constituent threedimensional models includes combining a mesh of the first mesh modelwith a mesh of the second mesh model.

Example 77

This example includes any or all of the features of example 75, wherein:the first constituent model is associated with first metadataidentifying one or more first anchor points in the first constituentmodel, each first anchor point corresponding to an access point in thefirst real world location; the second constituent model is associatedwith second metadata identifying one or more second anchor points in thesecond constituent model, each second anchor point corresponding to anaccess point in the first real world location; and combining theplurality of constituent three dimensional models to produce theaggregate model includes joining the first and second constituent modelssuch that in the aggregate model, a first anchor point of the firstconstituent model is substantially aligned with a second anchor point ofthe second constituent model.

Example 78

This example includes any or all of the features of example 77, wherein:the first constituent model includes a plurality of first anchor points;and in the aggregate model, a selected one of the plurality of firstanchor points is substantially aligned with a second anchor point of thesecond constituent model.

Example 79

This example includes any or all of the features of example 78, wherein:each of the plurality of first anchor points of the first constituentmodel is associated with first contextual information; each secondanchor point of the second constituent model is associated with secondcontextual information; and the instructions when executed furtherresult in the performance of the following operations including:identifying the selected one of the plurality of first anchor pointsbased at least on a comparison of the first contextual information ofeach of the plurality of first anchor points with second contextualinformation associated with each second anchor point.

Example 80

This example includes any or all of the features of example 79, whereinthe contextual information includes location information.

Example 81

This example includes any or all of the features of example 80, whereinthe location information specifies a real world location of each of thefirst and second anchor points.

Example 82

This example includes any or all of the features of example 81, whereinidentifying the selected one of the plurality of first anchor pointsincludes determining whether a first anchor point is within a thresholddistance of a second anchor point.

Example 83

This example includes any or all of the features of example 79, whereinthe contextual information further includes orientation information, theorientation information specifying an orientation of a corresponding oneof the first and second anchor points.

Example 84

This example includes any or all of the features of example 83, whereinidentifying the selected one of the plurality of first anchor pointsincludes determining whether a first anchor point is within a thresholddistance of a second anchor point.

Example 85

This example includes any or all of the features of example 84, whereinthe instructions when executed further result in the performance of thefollowing operations including: determining whether the orientation ofthe selected one of the plurality of first anchor points is the same asthe second anchor point and, if not, reorienting at least one of thefirst constituent model and the second constitute model.

Example 86

This example includes any or all of the features of example 80, whereinidentifying the selected one of the plurality of first anchor points isperformed such that in the aggregate model, one or more of the pluralityof constituent three dimensional models does not conflict with anotherof the plurality of constituent three dimensional models.

Example 87

This example includes any or all of the features of example 75, whereinthe aggregate model is a video game world, and the instructions whenexecuted further result in the performance of the following operationsincluding: introducing video game characters into the video game world.

Example 88

According to this example there is provided an apparatus including meansto perform the method of any one of examples 30 to 58.

Example 89

According to this example there is provided a content creation deviceincluding: processing means; memory means; camera means; sensor means;and content creation means, wherein: the camera means is to obtain depthdata of a real world environment and color data of the real worldenvironment; and the content creation means is to: produce a threedimensional model of the real world environment based at least in parton the depth data; analyze at least one of the depth data and color datato detect one or more access points in the real world environment; andcorrelate a portion of the three dimensional model with a detectedaccess point.

The terms and expressions which have been employed herein are used asterms of description and not of limitation, and there is no intention,in the use of such terms and expressions, of excluding any equivalentsof the features shown and described (or portions thereof), and it isrecognized that various modifications are possible within the scope ofthe claims. Accordingly, the claims are intended to cover all suchequivalents. Various features, aspects, and embodiments have beendescribed herein. The features, aspects, and embodiments are susceptibleto combination with one another as well as to variation andmodification, as will be understood by those having skill in the art.The present disclosure should, therefore, be considered to encompasssuch combinations, variations, and modifications.

What is claimed is:
 1. A content creation device comprising a processor,a memory, a camera module, at least one sensor, and a content creationmodule (CCM), wherein: the camera module comprises a depth camera toobtain depth data of a real world environment and a visual camera toobtain color data of the real world environment; the at least one sensoris to obtain contextual information; and said CCM comprises circuitryto: produce a three dimensional model of said real world environmentbased at least in part on said depth data; analyze at least one of saiddepth data and color data to detect one or more access points in saidreal world environment; correlate a portion of said three dimensionalmodel with a detected access point; and correlate said contextualinformation with the portion of said three dimensional modelcorresponding to a detected access point.
 2. The content creation deviceof claim 1, wherein said at least one sensor comprises one or more of agyroscope, a global positioning sensor, or an accelerometer.
 3. Thecontent creation device of claim 1, wherein: said depth data is storedin one or more first frames in said memory; said color data is stored inone or more second frames in said memory; and said CCM is to detect saidone or more access points at least in part by executing image processingoperations on at least one of said first or second frames.
 4. Thecontent creation device of claim 3, wherein: said image processingoperations comprise performing object detection on at least one of saidfirst or second frames; said CCM is further to identify a type of saiddetected access point at least in part by comparing an output of saidobject detection operations to a database of predefined access points.5. The content creation device of claim 1, wherein: said depth data isstored in one or more first frames in said memory; said color data isstored in one or more second frames in said memory; said CCM is furtherto execute image processing operations on at least one of said first orsecond frames to detect one or more objects in said real worldenvironment; said three dimensional model includes a representation of adetected object within said real world environment.
 6. The contentcreation device of claim 5, wherein: said image processing operations todetect one or more objects comprise performing object detection on atleast one of said first or second frames, and said CCM is to identify atype of detected object at least in part by comparing an output of saidimage processing operations to a database of predefined objects; saidCCM is further to: determine object parameters corresponding to anidentified detected object; and correlate said object parameters with arepresentation of an identified detected object in said threedimensional model.
 7. The content creation device of claim 1, wherein:said CCM is further to combine a plurality of constituent threedimensional models to produce an aggregate model, the plurality ofconstituent three dimensional models comprising a first constituentmodel of a first real world location and a second constituent model of asecond real world location, the first and second real world locationsbeing arranged in a real world arrangement; and in said aggregate model,said first and second constituent models are arranged in substantiallythe same manner as the real world arrangement.
 8. The content creationdevice of claim 7, wherein: said first constituent model is associatedwith first metadata identifying one or more first anchor points in saidfirst constituent model, each first anchor point corresponding to anaccess point in said first real world location; said second constituentmodel is associated with second metadata identifying one or more secondanchor points in said second constituent model, each second anchor pointcorresponding to an access point in said first real world location; andsaid CCM is further to produce said aggregate model at least in part byjoining at said first and second constituent models, such that in saidaggregate model a first anchor point of said first constituent model issubstantially aligned with a second anchor point of said secondconstituent model.
 9. The content creation device of claim 8, wherein:said first constituent model comprises a plurality of first anchorpoints; and said CCM is further to combine said plurality of constituentmodels such that in said aggregate model, a selected one of theplurality of first anchor points is substantially aligned with a secondanchor point of said second constituent model.
 10. A method forproducing a three dimensional model, comprising, with a content creationdevice: producing a three dimensional model of a real world environmentbased at least in part on depth data of said real world environment;analyzing at least one of said depth data and color data of said realworld environment to detect one or more access points in said real worldenvironment; correlating a portion of said three dimensional model witha detected access point; and correlating contextual information of atleast one sensor with the portion of said three dimensional modelcorresponding to said detected access point.
 11. The method of claim 10,further comprising: obtaining said depth data in one or more firstframes with a depth camera; and obtaining said color data with one ormore second frames with a visual camera; wherein: said analyzing atleast one of said depth data and color data to detect one or more accesspoints comprises executing image processing operations on at least oneof said first or second frames; and said three dimensional modelincludes a representation of a detected access point.
 12. The method ofclaim 11, further comprising: performing object detection operations onat least one of said first or second frames to detect objects in saidreal world environment, and identifying a type of detected object atleast in part by comparing an output of said object detection operationsto a database of predefined access points.
 13. The method of claim 10,further comprising: analyzing at least one of said depth data and colordata to detect an object within said real world environment; whereinsaid analyzing comprises executing image processing operations on atleast one of said first or second frames.
 14. The method of claim 13,wherein: said image processing operations comprise performing objectdetection on at least one of said first or second frames to detect oneor more detected objects; and the method further comprises: identifyinga detected object at least in part by comparing an output of said imageprocessing operations to a database of predefined objects; determiningobject parameters corresponding to an identified detected object; andcorrelating said object parameters with a representation of anidentified detected object in said three dimensional model.
 15. Themethod of claim 10, further comprising, with the content creationdevice: combining a plurality of constituent three dimensional models toproduce an aggregate model, the plurality of constituent threedimensional models comprising a first constituent model of a first realworld location and a second constituent model of a second real worldlocation, the first and second real world locations being arranged in areal world arrangement; wherein in said aggregate model, said first andsecond constituent models are arranged in substantially the same manneras the real world arrangement.
 16. The method of claim 15, wherein: saidfirst constituent model is associated with first metadata identifyingone or more first anchor points in said first constituent model, eachfirst anchor point corresponding to an access point in said first realworld location; said second constituent model is associated with secondmetadata identifying one or more second anchor points in said secondconstituent model, each second anchor point corresponding to an accesspoint in said first real world location; and combining said plurality ofconstituent three dimensional models to produce said aggregate modelcomprises joining said first and second constituent models such that insaid aggregate model, a first anchor point of said first constituentmodel is substantially aligned with a second anchor point of said secondconstituent model.
 17. At least one computer readable medium comprisinginstructions which when executed by a processor of a content creationdevice result in the performance of the following operations comprising:producing a three dimensional model of a real world environment based atleast in part on depth data of said real world environment; analyzing atleast one of said depth data and color data of said real worldenvironment to detect one or more access points in said real worldenvironment; correlating a portion of said three dimensional model witha detected access point; and correlating contextual information of atleast one sensor with the portion of said three dimensional modelcorresponding to said detected access point.
 18. The at least onecomputer readable medium of claim 17, wherein said instructions whenexecuted further result in the performance of the following operationscomprising: obtaining said depth data in one or more first frames with adepth camera; and obtaining said color data with one or more secondframes with a visual camera; wherein: said analyzing at least one ofsaid depth data and color data to detect one or more access pointscomprises executing image processing operations on at least one of saidfirst or second frames to detect access points within said real worldenvironment; and said three dimensional model includes a representationof a detected access point.
 19. The at least one computer readablemedium of claim 18, wherein said instructions when executed furtherresult in the performance of the following operations comprising:performing object detection operations on at least one of said first orsecond frames to detect objects in said real world environment, andidentifying a type of detected object at least in part by comparing anoutput of said object detection operations to a database of predefinedaccess points.
 20. The at least one computer readable medium of claim19, wherein said instructions when executed further result in theperformance of the following operations comprising determining objectparameters corresponding to an identified detected object; andcorrelating said object parameters with a representation of anidentified detected object in said three dimensional model.
 21. The atleast one computer readable medium of claim 17, wherein saidinstructions when executed further result in the performance of thefollowing operations comprising: combining a plurality of constituentthree dimensional models to produce an aggregate model, the plurality ofconstituent three dimensional models comprising a first constituentmodel of a first real world location and a second constituent model of asecond real world location, the first and second real world locationsbeing arranged in a real world arrangement; wherein in said aggregatemodel, said first and second constituent models are arranged insubstantially the same manner as the real world arrangement.
 22. The atleast one computer readable medium of claim 21, wherein: said firstconstituent model is associated with first metadata identifying one ormore first anchor points in said first constituent model, each firstanchor point corresponding to an access point in said first real worldlocation; said second constituent model is associated with secondmetadata identifying one or more second anchor points in said secondconstituent model, each second anchor point corresponding to an accesspoint in said first real world location; and combining said plurality ofconstituent three dimensional models to produce said aggregate modelcomprises joining said first and second constituent models such that insaid aggregate model, a first anchor point of said first constituentmodel is substantially aligned with a second anchor point of said secondconstituent model.
 23. The at least one computer readable medium ofclaim 22, wherein: said first constituent model comprises a plurality offirst anchor points; and in said aggregate model, a selected one of theplurality of first anchor points is substantially aligned with a secondanchor point of said second constituent model.
 24. The at least onecomputer readable medium of claim 23, wherein: each of the plurality offirst anchor points of said first constituent model is associated withfirst contextual information; each second anchor point of the secondconstituent model is associated with second contextual information; andsaid instructions when executed further result in the performance of thefollowing operations comprising: identifying said selected one of theplurality of first anchor points based at least on a comparison of thefirst contextual information of each of said plurality of first anchorpoints with second contextual information associated with each secondanchor point.
 25. The at least one computer readable medium of claim 21,wherein said aggregate model is a video game world, and saidinstructions when executed further result in the performance of thefollowing operations comprising: introducing video game characters intosaid video game world.